Method of controlling mobile robot, apparatus for supporting the method, and delivery system using mobile robot

ABSTRACT

Provided are a method of controlling a mobile robot, apparatus for supporting the method, and delivery system using the mobile robot. The method, which is performed by a control apparatus, comprises acquiring a first control value for the mobile robot, which is input through a remote control apparatus, acquiring a second control value for the mobile robot, which is generated by an autonomous driving module, determining a weight for each control value based on a delay between the mobile robot and the remote control apparatus and generating a target control value of the mobile robot in combination of the first control value and the second control value based on the determined weights, wherein a first weight for the first control value and a second weight for the second control value are inversely proportional to each other.

CROSS-REFERENCE TO RELATED APPLICATIONS

This present application is a Divisional of U.S. patent application Ser.No. 16/199,990, filed on Nov. 26, 2018, which claims the benefit ofpriority from Korean Patent Application No. 10-2018-0022957 filed onFeb. 26, 2018, No. 10-2018-0046018 filed on Apr. 20, 2018 and No.10-2018-0133322 filed on Nov. 2, 2018 in the Korean IntellectualProperty Office, the disclosure of all of which are hereby incorporatedby reference in its entirety.

BACKGROUND 1. Field of the Disclosure

The present disclosure relates to a method of controlling a mobilerobot, an apparatus for supporting the control method, and a deliverysystem using a mobile robot, and more particularly, to a method ofcontrolling a mobile robot designed to solve a problem that causes anincrease in a risk of accident of a mobile robot and a problem thatcauses a user's sense of difference in operation due to a communicationdelay between a remote control system and a mobile robot, a controlapparatus for supporting the control method, and various deliverysystems using one or more mobile robots.

2. Description of the Related Art

Recently, interest in autonomous mobile robots such as a self-drivingvehicle has increased. However, research on autonomous mobile robots isstill in its infancy, and most robot systems include a remote controlsystem for an operator to control a mobile robot at a remote location.

In a system for controlling a mobile robot at a remote location throughthe remote control system, the most problematic is a communication delaythat may occur during a wireless communication process. As shown in FIG.1 , the communication delay includes a control delay 5 that occurs whilea mobile robot 3 receives a control signal of a remote control system 1and a monitoring delay 7 that occurs while the remote control system 1receives images of the surroundings of the mobile robot 3 from themobile robot 3.

The two delays 5 and 7 are main factors that increase a risk of accidentof the mobile robot. This is because a user may later recognize anobstacle near the mobile robot 3 because of the monitoring delay 7 and acontrol signal of an operator may later reach the mobile robot 3 becauseof the control delay 5.

Conventionally, the above problems have been mainly solved by changingthe driving mode of a mobile robot to an autonomous driving mode whenthe communication delay exceeds a threshold value. That is, when remotecontrol is not easy due to the communication delay, a method of avoidingthe risk of accident by giving authority to control the mobile robot toan autonomous driving module is mainly used. However, as shown in FIG. 2, iterative mode changes during the operation of the mobile robot maydiscontinuously generate a final control value 15. As a result, themobile robot may be unstably operated (e.g., a robot control valuechanges suddenly at times T1, T2, and T3) and thus a user may lose hisor her attention. Furthermore, the sudden change to the autonomousdriving mode may cause an operator's sense of difference in robotcontrol to be maximized.

Accordingly, to control a mobile robot at a remote location, a mobilerobot control method capable of solving problems about a mobile robot'ssafety and a user's sense of difference is required.

Meanwhile, recently, as technology for ground robots or drones evolves,many attempts have been made to apply such technology to deliveryservices. For example, Google is planning to utilize unmanned cars fordelivery, and Amazon is trying to make a drones-based delivery servicecalled “Prime Air.” Also, Domino's, which is a global pizza deliverycompany, has demonstrated a service for delivering pizza using a robot.The purpose of such a delivery service is to improve customers'satisfaction by delivering goods to users more quickly and to increasedelivery efficiency by reducing labor costs.

However, much research still has to be done in order for a deliveryservice using ground robots and drones to be put to practical use invarious environments. For example, in order for the delivery service tobe put to practical use, research should be done on various topics suchas a collaborative delivery method using ground robots and drones, alarge-size item delivery method using a plurality of drones, and amethod of safely delivering a fragile item such as food.

SUMMARY

Aspects of the present disclosure provide a control method for a mobilerobot and an apparatus for supporting the control method, the controlmethod being capable of solving a problem of increasing a risk ofaccident of a mobile robot, which occurs due to a communication delay.

Aspects of the present disclosure also provide a control method for amobile robot and an apparatus for supporting the control method, thecontrol method being capable of solving a problem of a user's sense ofdifference in robot operation due to a communication delay and/oriterative mode changes.

Aspects of the present disclosure also provide a system for providing adelivery service using a mobile robot and a method of operating thesystem.

Aspects of the present disclosure also provide a system for providing acollaborative delivery service using both of a ground robot and a droneand a method of operating the system.

Aspects of the present disclosure also provide a method of safelydelivering a large item using a plurality of drones.

Aspects of the present disclosure also provide a method of safelydelivering a fragile item such as food.

It should be noted that objects of the present disclosure are notlimited to the above-described objects, and other objects of the presentdisclosure will be apparent to those skilled in the art from thefollowing descriptions.

According to an aspect of the present disclosure, there is provided amethod of controlling a mobile robot. The method being performed by acontrol apparatus comprises acquiring a first control value for themobile robot, which is input through a remote control apparatus,acquiring a second control value for the mobile robot, which isgenerated by an autonomous driving module, determining a weight for eachcontrol value based on a delay between the mobile robot and the remotecontrol apparatus and generating a target control value of the mobilerobot in combination of the first control value and the second controlvalue based on the determined weights, wherein a first weight for thefirst control value and a second weight for the second control value areinversely proportional to each other.

According to another aspect of the present disclosure, there is provideda collaborative delivery system using a mobile robot. The collaborativedelivery system comprises a service system configured to generatemission information for delivering an item to a delivery destination, afirst mobile robot configured to transport the item based on the missioninformation and a second mobile robot configured to transport the itemin collaboration with the first mobile robot, wherein the missioninformation includes collaboration information necessary for the firstmobile robot and the second mobile robot to collaborate with each other

According to still another aspect of the present disclosure, there isprovided a food delivery system using a mobile robot. The food deliverysystem comprises a service system configured to receive a deliveryrequest from a user terminal and generate mission information fordelivering food to a delivery destination in response to the deliveryrequest and a mobile robot configured to deliver the food contained in acooking appliance based on the mission information, wherein the missioninformation includes a cooking time for the food, and wherein the mobilerobot calculates an expected time taken to reach the deliverydestination, compares the calculated expected time to the cooking time,and operates the cooking appliance based on a result of the comparison.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects and features of the present disclosure willbecome more apparent by describing in detail exemplary embodimentsthereof with reference to the attached drawings, in which:

FIG. 1 is a diagram illustrating a communication delay between a remotecontrol system and a mobile robot;

FIG. 2 is a diagram illustrating problems of a conventional mobile robotcontrol scheme;

FIG. 3 is an example configuration diagram showing a robot controlsystem according to some embodiments of the present disclosure;

FIGS. 4 and 5 are diagrams schematically illustrating a controlapparatus according to some embodiments of the present disclosure;

FIG. 6 is a diagram illustrating operation of a server according to someembodiments of the present disclosure;

FIG. 7 is an example block diagram showing an apparatus for controllinga mobile robot according to some embodiments of the present disclosure;

FIG. 8 is an example flowchart showing a method of controlling a mobilerobot according to a first embodiment of the present disclosure;

FIGS. 9 and 10 are diagrams illustrating a weight for each control valueto be referenced according to some embodiments of the presentdisclosure;

FIG. 11 is an example flowchart showing a weight determination processaccording to some embodiments of the present disclosure;

FIG. 12 is an example flowchart showing a weight determination processaccording to other embodiments of the present disclosure;

FIG. 13 is an example flowchart showing a method of controlling a mobilerobot according to a second embodiment of the present disclosure;

FIGS. 14 and 15 are example diagrams showing a driving pattern learningmethod according to some embodiment of the present disclosure;

FIG. 16 is a diagram illustrating a delivery system using a mobile robotaccording to some embodiments of the present disclosure;

FIG. 17 is an example configuration diagram showing a service systemaccording to some embodiments of the present disclosure;

FIG. 18 is an example flowchart showing a mobile robot-based deliveryservice providing method that may be performed by a delivery systemaccording to some embodiments of the present disclosure;

FIG. 19 is an example diagram illustrating a method of identifying auser and a user terminal according to some embodiments of the presentdisclosure;

FIG. 20 is an example diagram illustrating a collaborative deliverymethod according to a first embodiment of the present disclosure;

FIG. 21 is an example diagram illustrating a collaborative deliverymethod according to a second embodiment of the present disclosure;

FIGS. 22 and 23 are example diagrams illustrating a collaborativedelivery method according to a third embodiment of the presentdisclosure;

FIGS. 24 and 25 are example diagrams illustrating a collaborativedelivery method according to a fourth embodiment of the presentdisclosure;

FIG. 26 is an example diagram illustrating a collaborative deliverymethod according to a fifth embodiment of the present disclosure;

FIG. 27 is an example diagram illustrating a collaborative deliverymethod according to a sixth embodiment of the present disclosure;

FIG. 28 is an example diagram illustrating a collaborative deliverymethod based on multiple drones according to a first embodiment of thepresent disclosure;

FIG. 29 is an example diagram illustrating a collaborative deliverymethod based on multiple drones according to a second embodiment of thepresent disclosure;

FIG. 30 is an example diagram illustrating a food delivery method usinga mobile robot according to a first embodiment of the presentdisclosure;

FIGS. 31 and 32 are example diagrams illustrating a food delivery methodusing a mobile robot according to a second embodiment of the presentdisclosure; and

FIG. 33 is an example hardware configuration diagram illustrating anexample computing apparatus capable of implementing an apparatus and/ora system according to various embodiments of the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Hereinafter, preferred embodiments of the present disclosure will bedescribed with reference to the attached drawings. Advantages andfeatures of the present disclosure and methods of accomplishing the samemay be understood more readily by reference to the following detaileddescription of preferred embodiments and the accompanying drawings. Thepresent disclosure may, however, be embodied in many different forms andshould not be construed as being limited to the embodiments set forthherein. Rather, these embodiments are provided so that this disclosurewill be thorough and complete and will fully convey the concept of thedisclosure to those skilled in the art, and the present disclosure willonly be defined by the appended claims. Like numbers refer to likeelements throughout.

Unless otherwise defined, all terms including technical and scientificterms used herein have the same meaning as commonly understood by one ofordinary skill in the art to which this disclosure belongs. Further, itwill be further understood that terms, such as those defined in commonlyused dictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art and thepresent disclosure, and will not be interpreted in an idealized oroverly formal sense unless expressly so defined herein. The terms usedherein are for the purpose of describing particular embodiments only andis not intended to be limiting. As used herein, the singular forms areintended to include the plural forms as well, unless the context clearlyindicates otherwise.

The terms “comprise”, “include”, “have”, etc. when used in thisspecification, specify the presence of stated features, integers, steps,operations, elements, components, and/or combinations of them but do notpreclude the presence or addition of one or more other features,integers, steps, operations, elements, components, and/or combinationsthereof.

Prior to describing the present specification, some terms used in thepresent specification will be described.

Herein, the term “mobile robot” collectively refers to any kinds ofrobots having a moving (that is, traveling) function. The mobile robotmay include a ground robot having a ground movement function and a dronehaving an air movement function. Also, the mobile robot may include aremote control robot controlled by a remote control device (or system),an autonomous robot moving according to autonomous determination, and asemi-autonomous robot having both of a remote control function and anautonomous moving function.

Herein, the term “instruction” refers to an element of a computerprogram and a series of commands that are grouped on a function basisand executed by a processor.

Hereinafter, some embodiments of the present disclosure will bedescribed in detail with reference to the accompanying drawings.

FIG. 3 is an example configuration diagram showing a robot controlsystem according to some embodiments of the present disclosure.

Referring to FIG. 3 , the robot control system may be configured toinclude a remote control system 20, a mobile robot 30, a controlapparatus 100 (not shown in FIG. 3 ), and a server 50. However, itshould be appreciated that this is merely an example embodiment forachieving the objects of the present disclosure and some elements areadded to, or deleted from, the configuration as needed. Also, elementsof the robot control system shown in FIG. 3 indicate functional elementsthat are functionally distinct from one another. It should be noted thatat least one element may be integrated with one another in an actualphysical environment. For example, the control apparatus 100 may beimplemented with a specific logic of the remote control system 20 or themobile robot 30.

Also, in an actual physical environment, each of the elements may beseparated into a plurality of sub-function elements. For example, theserver 50 may have a first function implemented in a first computingapparatus and a second function implemented in a second computingapparatus. Each element of the robot control system will be describedbelow.

In the robot control system, the remote control system 20 is anapparatus that is used by an operator to remotely control the mobilerobot 30. The remote control system 20 may control the mobile robot 30,which is located at a remote location, by transmitting a control value(e.g., a steering angle and a speed) input by the operator to the mobilerobot 30 in real time. For convenience of description, the control valueinput by the operator through the remote control system 20 may bereferred to as a “remote control value.”

To improve convenience, the remote control system 20 may be configuredto include a haptic interface device such as a steering wheel, a pedal,and the like. For example, the operator may control the steering angleof the mobile robot 30 through the steering wheel and may control thespeed of the mobile robot 30 through the pedal. However, the technicalscope of the present disclosure is not limited thereto, and a userinterface apparatus of the remote control system 20 may be implementedin an arbitrary manner.

Also, the remote control system 20 may be configured to include adisplay apparatus for displaying data regarding surroundings-relatedimages captured by the mobile robot 30. Accordingly, the operator mayrecognize surrounding environments of the mobile robot 30 through thedisplay apparatus and may suitably perform remote control.

In the robot control system, the mobile robot 30 is a robot that iscontrolled by the remote control system 20. In particular, FIG. 3 showsan example in which the mobile robot 30 is a vehicle-based ground robot.As described above, the mobile robot 30 may include any types of robotssuch as a drone.

In the robot control system, the control apparatus 100 is a computingapparatus that generates a target control value for controlling themobile robot 30 based on various control values. Here, the computingapparatus may be a notebook, a desktop, a laptop, or the like. However,the present disclosure is not limited thereto, and the computingapparatus may include any kinds of devices equipped with computingmeans. One example of the computing apparatus will be described withreference to FIG. 33 .

Since the target control value is a value that indicates a targetcontrol state (e.g., a steering angle and a speed) of the mobile robot30, a final control value input to the mobile robot 30 may be differentfrom the target control value. For example, the final control value maybe generated by various control algorithms based on the target controlvalue and a current control state (e.g., a current speed and a currentsteering angle) of the mobile robot 30. However, a detailed descriptionthereof will be omitted in order not to obscure the subject matters ofthe present disclosure.

As shown in FIG. 4 , the control apparatus 100 may generate a targetcontrol value 29 based on a remote control value 23 input through theremote control system 20, a control value 25 (hereinafter referred to asan “autonomous control value”) generated by an autonomous driving module21, and the like. In this case, the control apparatus 100 may adjust theproportions in which the remote control value 23 and the autonomouscontrol value 25 are reflected in the target control value 29, based onvarious factors such as a communication delay between the remote controlsystem 20 and the mobile robot 30, a risk of collision, and a level ofcomplexity of surrounding environments.

That is, the control apparatus 100 does not generate a target controlvalue using specific control values 23 and 25 at any point, butgenerates the target control value 29 by suitably combining the remotecontrol value 23 and the autonomous control value 25. Accordingly, asshown in FIG. 5 , the target control value 29 generated by the controlapparatus 100 appears in the form of a continuous value and does notfluctuate abruptly. Therefore, it is possible to solve a problem of adiscontinuous control value being generated due to iterative controlmode changes (see FIG. 2 ) and also to alleviate a problem of anoperator feeling a sense of difference in operation. A configuration andoperation of the control apparatus 100 will be described in detail withreference to FIG. 7 and subsequent drawings.

The control apparatus 100 may be implemented at a side of the mobilerobot 30, but the technical scope of the present disclosure is notlimited thereto. However, in order to facilitate understanding, thefollowing description assumes that the control apparatus 100 isimplemented at the side of the mobile robot 30.

In the robot control system, the server 50 is a computing apparatus thatreceives driving data from the remote control system 20 or the mobilerobot 30, learns the driving data, and builds a machine learning modelfor a user's driving pattern. Here, the computing apparatus may be anotebook, a desktop, a laptop, or the like. However, the presentdisclosure is not limited thereto, and the computing apparatus mayinclude any kind of device equipped with computing means andcommunication means.

The driving data may include, for example, control values for remotecontrol systems 20 a to 20 n, sensing data (e.g., image data) of themobile robot, and the like.

In detail, as shown in FIG. 6 , the server 50 may receive driving dataof a specific user from the plurality of remote control systems 20 a to20 n, learn the driving data, and build a machine learning modelassociated with a driving pattern of each user. Also, the server 50 maypredict and provide a control value (hereinafter referred to as a“pattern control value”) for the current driving environment of themobile robot 30 through the built machine learning model. The patterncontrol value may also be used by the control apparatus 100 to generatethe target control value, and a detailed description thereof will bedescribed below with reference to FIG. 12 . Also, an example method inwhich the server 50 learns the driving pattern will be described belowwith reference to FIG. 12 .

As reference, some functions of the server 50 may be implemented in theremote control system 20 or the mobile robot 30. For example, the remotecontrol system 20 or the mobile robot 30 may perform machine learning ona driving pattern of a specific user based on driving data and provide amachine learning model that learns the driving pattern to the server 50.In this case, the server 50 may perform only a function of providing apredetermined pattern control value to the control apparatus 100 usingthe machine learning model that learns driving patterns of a pluralityof users.

At least some of the elements 20, 30, 50, and 100 of the robot controlsystem may communicate over a network. A network between the remotecontrol system 20 and the mobile robot 30 may be typically implementedwith a wireless network, but the technical scope of the presentdisclosure is not limited thereto. That is, the network may beimplemented with any type of network such as a local area network (LAN),a wide area network (WAN), a mobile radio communication network, and awireless broadband Internet (WiBro).

The robot control system according to an embodiment of the presentdisclosure has been described with reference to FIGS. 3 to 6 .Subsequently, the configuration and operation of the control apparatus100 of the mobile robot according to an embodiment of the presentdisclosure will be described with reference to FIG. 7 .

FIG. 7 is an example block diagram showing the control apparatus 100according to some embodiments of the present disclosure.

Referring to FIG. 7 , the control apparatus 100 may be configured toinclude a control value acquisition module 110, a delay determinationmodule 120, an autonomous driving module 130, an information acquisitionmodule 140, a weight determination module 150, and a target controlvalue determination module 160. However, only elements related to theembodiment of the present disclosure are shown in FIG. 7 . Accordingly,it is to be understood by those skilled in the art that general-purposeelements other than the elements shown in FIG. 7 may be furtherincluded. Each element will be described below.

The control value acquisition module 110 acquires various kinds ofcontrol values serving as the basis of the target control value. Indetail, the control value acquisition module 110 may acquire a remotecontrol value input by a user from the remote control system 20 and mayacquire an autonomous control value generated by the autonomous drivingmodule 130. Each of the remote control value and the autonomous controlvalue may include control values for a speed, a steering angle, and thelike of the mobile robot 30.

Also, the control value acquisition module 110 may further acquire, fromthe server 50, a pattern control value obtained based on a drivingpattern of a corresponding user, a pattern control value obtained basedon an average driving pattern of a plurality of users, and the like.

Subsequently, the delay determination module 120 determines acommunication delay between the remote control system 20 and the mobilerobot 30. The communication delay may be determined based on the controldelay 5 and the monitoring delay 7 as shown in FIG. 1 . For example, thecommunication delay may be determined in various forms such as the sum,the average, and the weighted average of the two delays 5 and 7.

Subsequently, the autonomous driving module 130 generates an autonomouscontrol value for the mobile robot 30. To this end, the autonomousdriving module 130 may use at least one autonomous driving algorithmwell-known in the art, and any autonomous algorithm may be used.

Subsequently, the information acquisition module 140 acquires variouskinds of information such as a risk of collision with an object near themobile robot 30, a level of complexity of surrounding environments, etc.The risk of collision and the level of complexity may be used by theweight determination module 150 to determine a weight for each controlvalue.

The information acquisition module 140 may directly calculate the riskof collision and the level of complexity and may receive informationcalculated by another apparatus. This may depend on the embodiment. Amethod in which the information acquisition module 140 determines alevel of complexity and a risk of collision with a nearby object basedon sensing data of the mobile robot will be described with reference toFIG. 12 .

Subsequently, the weight determination module 150 determines a weightfor each control value and provides the determined weight to the targetcontrol value determination module 160. In this case, the weight may beunderstood as a value indicating the proportion in which each controlvalue is reflected in the target control value.

In some embodiments, the weight determination module 150 may determine aweight for each control value based on a communication delay provided bythe delay determination module 120. A detailed description of thisembodiment will be described in detail with reference to FIGS. 10 and 11.

In some embodiments, the weight determination module 150 may determine aweight for each control value in further consideration of the risk ofcollision with an object near the mobile robot 30, the risk of collisionbeing provided by the information acquisition module 140. A detaileddescription of this embodiment will be described below with reference toFIG. 12 .

In some embodiments, the weight determination module 150 may determine aweight for each control value in further consideration of the level ofcomplexity of surrounding environments of a mobile robot 30, the levelof complexity being provided by the information acquisition module 140.A detailed description of this embodiment will be described below withreference to FIG. 12 .

Subsequently, the target control value determination module 160determines a target control value based on a plurality of control valuesprovided by the control value acquisition module 110 and a weight foreach control value provided by the weight determination module 150.

For example, the target control value determination module 160 maydetermine the target control value based on the weighted average of theplurality of control values. However, this is merely illustrative ofsome embodiments of the present disclosure, and the technical scope ofthe present disclosure is not limited thereto.

It should be noted that not all elements shown in FIG. 7 are essentialelements of the control apparatus 100. That is, according to anotherembodiment of the present disclosure, the control apparatus 100 may beimplemented with some of the elements shown in FIG. 7 .

Each of the elements 110 to 160 shown in FIG. 7 may denote software orhardware such as a field programmable gate array (FPGA) or anapplication specific integrated circuit (ASIC). However, the elementsare not limited to software or hardware, and may be configured to be inan addressable storage medium or configured to activate one or moreprocessors. Functions provided by the elements may be implemented withsub-elements and may be implemented with one element that performs aspecific function by combining the plurality of elements.

The configuration and operation of the control apparatus 100 accordingto some embodiments of the present disclosure have been described withreference to FIG. 7 . The control method for the mobile robot accordingto some embodiments of the present disclosure will be described below indetail below with reference to FIGS. 8 to 12 .

Each step of the control method for the mobile robot according to someembodiments of the present disclosure, which will be described below,may be performed by a computing apparatus. That is, each step of thecontrol method may be implemented with one or more instructions executedby a processor of the computing apparatus. All the steps of the controlmethod may be executed by a single physical computing apparatus.However, first steps of the method may be performed by a first computingapparatus, and second steps of the method may be performed by a secondcomputing apparatus. The following description assumes that each step ofthe control method is performed by the control apparatus 100. However,for convenience, the description of the operating subject of each stepincluded in the control method may be omitted.

FIG. 8 is an example flowchart showing a control method for a mobilerobot according to a first embodiment of the present disclosure.However, it should be appreciated that this is merely an exampleembodiment for achieving the objects of the present disclosure and somesteps are added to, or deleted from, the configuration as needed.

Referring to FIG. 8 , the control method for the mobile robot accordingto the first embodiment begins with step S100 in which the controlapparatus 100 acquires a remote control value that is input through theremote control system 20. In this case, the remote control value mayinclude control values for a speed, a steering angle, and the like ofthe mobile robot 30.

In step S200, the control apparatus 100 acquires an autonomous controlvalue generated by the autonomous driving module. The autonomous controlvalue may also include control values for a speed, a steering angle, andthe like of the mobile robot 30.

In step S300, the control apparatus 100 determines a weight for eachcontrol value.

In some embodiments, the control apparatus 100 may determine a weightfor each control value based on a communication delay between the remotecontrol system 20 and the mobile robot 30. A detailed example of thisembodiment is shown in FIG. 9 .

As shown in FIG. 9 , a weight 41 for a remote control value (hereinafterreferred to as a “remote weight”) and a weight 43 for an autonomouscontrol value (hereinafter referred to as an “autonomous weight”) may bedetermined as values that are inversely proportional to each other. Indetail, as the communication delay increases, the remote weight 41 maybe determined as a larger value, and the autonomous weight 43 may bedetermined as a smaller value. This is to improve the safety of themobile robot 30 by increasing the proportion in which the autonomouscontrol value is reflected in the target control value as thecommunication delay increases.

Also, the control apparatus 100 may determine the autonomous weight 43as the maximum value and determine the remote weight 41 as the minimumvalue in response to determination that the communication delay isgreater than or equal to a predetermined threshold value 45. Forexample, as shown in FIG. 9 , when the weight for each control value hasa value from 0 to 1, the control apparatus 100 may determine theautonomous weight 43 to be “1” and may determine the remote weight 41 tobe “0.” In this case, the mobile robot 30 may be controlled as if it isoperating in an autonomous driving mode.

The threshold value 45 may be a predetermined constant or a variablevarying depending on situations. For example, the threshold value 45 maybe a variable varying according to a current traveling speed of themobile robot 30. In detail, for example, the threshold value 45 may bedetermined to be a value that is inversely proportional to the currenttraveling speed of the mobile robot 30, as shown in FIG. 10 , and mayvary as the current traveling speed of the mobile robot 30 changes. Inthis case, when the current traveling speed of the mobile robot 30 ishigher even through the communication delay is maintained, the controlapparatus 100 determines the autonomous weight 43 to be a lager value.Accordingly, it is possible to further decrease the risk of accident ofthe mobile robot 30 and to improve safety. However, this is merelyillustrative of some embodiments of the present disclosure, and thetechnical scope of the present disclosure is not limited thereto.

In some embodiments, as shown in FIG. 11 , the control apparatus 100 maydetermine a weight for each control value in further consideration of arisk of collision with an object near the mobile robot 30 (S310, S330,and S350). For example, in order to improve the safety of the mobilerobot 30, the control apparatus 100 may determine the autonomous weightto be a larger value as the risk of collision increases.

The risk of collision may be determined based on sensing data of themobile robot 30. In this case, the sensing data may include datameasured by various kinds of sensors, for example, data regarding adistance from a nearby object which is measured through a proximitysensor and data regarding an image which is captured by a camera sensor.

According to some embodiments of the present disclosure, the controlapparatus 100 may determine the risk of collision based on an imageanalysis result such as the number of nearby objects, the result ofrecognizing a nearby object, a distance from the nearby object, and thecharacteristics (or features) of surrounding environments (e.g., a flatroad, an uphill road, a downhill road, the presence of lanes, the sizeof lanes, the presence of intersections, the presence of traffic lights,etc.), which are obtained from the image data. In this case, the resultof recognition may include information regarding whether the nearbyobject is a mobile object or a stationery object, whether the nearbyobject is a person or thing, and the like and information regarding thesize, material, and the like of the corresponding object.

The method in which the control apparatus 100 determines the risk ofcollision with the nearby object may vary depending on the embodiment.For example, the risk of collision may be determined to be a highervalue as the number of nearby objects located within a certain distanceincreases. As another example, the risk of collision may be determinedto be a higher value as the number of nearby objects corresponding tomobile objects within a certain distance increases. As still anotherexample, the risk of collision may be determined to be a higher value asthe number of nearby objects corresponding to persons within a certaindistance increases. This is because a person is likely to show a moreirregular movement pattern than a thing. As still another example, whena plurality of nearby objects are located within a certain distance, butmost of the corresponding nearby objects are located in other lanes, therisk of collision may be determined to be a lower value. As stillanother example, the risk of collision may be determined to be a highervalue as the level of complexity of surrounding environments increases.As still another example, the risk of collision may be determined to bea higher value as the size of the nearby object and the hardness of thematerial of the nearby object increase.

In some embodiments, as shown in FIG. 12 , the control apparatus 100 maydetermine a weight for each control value in further consideration of alevel of collision of surrounding environments of the mobile robot 30(S310, S320, and S330).

The level of complexity may also be determined based on sensing data ofthe mobile robot 30. In this case, the sensing data may include datameasured by various kinds of sensors, for example, data regarding adistance from a nearby object which is measured through a proximitysensor and data regarding an image which is captured by a camera sensor.

According to some embodiments of the present disclosure, the controlapparatus 100 may determine the level of complexity based on an imageanalysis result such as the number of nearby objects, the result ofrecognizing a nearby object, a distance from the nearby object, and thecharacteristics of surrounding environments (e.g., the presence oflanes, the presence of intersections, the size of lanes, the presence oftraffic lights, etc.), a location distribution of the nearby objects,and a density of the nearby objects, which are obtained from the imagedata. For example, as the number of nearby objects increases, the levelof complexity may be determined to be a higher value. As anotherexample, as the number of mobile objects increases, the level ofcomplexity may be determined to be a higher value. As still anotherexample, as the number of mobile objects showing irregular movementpatterns increases, the level of complexity may be determined to be ahigher value. As still another example, as the number of types of nearbyobjects increases, the level of complexity may be determined to be ahigher value. In addition, the level of complexity may be determined tobe a higher value when the surrounding environments indicate that thereis an intersection, that a lane is small, or that there is no lane orwhen a plurality of nearby objects are distributed within a certaindistance.

In addition, the control apparatus 100 may determine a weight for eachcontrol value through various combinations of the aforementionedembodiments.

Referring to FIG. 8 again, in step S400, the control apparatus 100generates a target control value of the mobile robot based on a remotecontrol value, an autonomous control value, and a weight for eachcontrol value. For example, the control apparatus 100 may generate thetarget control value using a technique such as a weight sum, a weightedaverage, and the like.

In some embodiments, the control apparatus 100 may further perform astep of adjusting the target control value so that a difference betweenthe target control value and the remote control value is smaller than orequal to a predetermined threshold value. For example, the controlapparatus 100 may adjust the target control value to be within thethreshold value in response to determination that the difference betweenthe target control value and the remote control value in step S400 isgreater than or equal to the threshold value.

In this case, the threshold value may be a predetermined constant or avariable varying depending on situations. For example, the thresholdvalue may be a variable that is determined to be a larger value as thecurrent traveling speed or the communication delay of the mobile robot30 increases. This is because the weight for the remote control valuemay decrease as the current traveling speed or the communication delayincreases. According to this embodiment, the target control value may besuitably adjusted so that the difference between the target controlvalue and the remote control value does not increase excessively. Thus,it is possible to further decrease a user's sense of difference inremote operation.

The control method for the mobile robot according to the firstembodiment of the present disclosure has been described with referenceto FIGS. 8 to 12 . According to the above-described method, the targetcontrol value is generated based on a combination of the control values.Thus, it is possible to alleviate a problem of the operation of themobile robot being unstable due to iterative control mode changes and aproblem of an operator's sense of difference in operation beingmaximized. Furthermore, by adjusting a weight for each control value, itis possible to minimize the risk of accident of the mobile robot evenwithout the control mode switching.

The control method for the mobile robot according to the secondembodiment of the present disclosure will be described with reference toFIGS. 13 to 15 . In order to exclude redundant description, thefollowing description focuses on a difference from the aforementionedfirst embodiment. However, it will be appreciated that the contentsmentioned in the first embodiment may be applied to the secondembodiment.

FIG. 13 is an example flowchart showing a control method for a mobilerobot according to the second embodiment of the present disclosure.However, it should be appreciated that this is merely an exampleembodiment for achieving the objects of the present disclosure and somesteps are added to, or deleted from, the configuration as needed.

Referring to FIG. 13 , like the first embodiment, the control methodaccording to the second embodiment begins with steps in which thecontrol apparatus 100 acquires a remote control value and an autonomouscontrol value (S100, S200).

In step S250, the control apparatus 100 acquires a pattern control valuefor the mobile robot 30. As described above, the pattern control valuerefers to a control value generated by a machine learning model thatlearns a user's driving pattern. That is, the pattern control valuerefers to a control value that is predicted for the current surroundingenvironment of the mobile robot 30 based on past driving patterns. Amethod of learning a driving pattern will be described below withreference to FIGS. 14 and 15 .

The pattern control value may include a first pattern control valuegenerated based on a driving pattern of a specific user who controls themobile robot 30 through the remote control system 20 and/or a secondpattern control value generated based on an average driving pattern of aplurality of users. The pattern control value may be received, forexample, from a predetermined server (e.g., the server 50), but thetechnical scope of the present disclosure is not limited thereto.

In step S350, the control apparatus 100 determines a weight for eachcontrol value. For example, the weight for the pattern control value maybe determined to be a larger value as the communication delay or thecurrent traveling speed of the mobile robot 30 increases.

In step S450, the control apparatus 100 generates a target control valueof the mobile robot 30 based on a remote control value, an autonomouscontrol value, a pattern control value, and a weight for each controlvalue. For example, the control apparatus 100 may generate the targetcontrol value using a technique such as a weight sum, a weightedaverage, and the like.

The control method for the mobile robot according to the secondembodiment of the present disclosure has been described with referenceto FIG. 13 . According to the above-described method, the target controlvalue is generated in further consideration of a pattern control valuegenerated based on a driving pattern of a corresponding operator ordriving patterns of a plurality of operators. When the pattern controlvalue corresponding to the driving pattern of the corresponding operatoris reflected in the target control value, it is possible to alleviate anoperator's sense of difference in operation. Also, when the patterncontrol value corresponding to the driving patterns of the plurality ofoperators is reflected in the target control value, an error of anothercontrol value may be corrected by the pattern control value, and thus itis possible to further improve the safety of the mobile robot.

The driving pattern learning method according to some embodiments of thepresent disclosure will be described below with reference to FIGS. 14and 15 . Each step of the driving pattern learning method to bedescribed below may be performed by a computing apparatus. For example,the computing apparatus may be a server 50, a control apparatus 100, aremote control system 20, a mobile robot 30, or the like. However, inorder to facilitate understanding, the following description assumesthat the operating subject of each step included in the driving patternlearning method is the server 50. For convenience of description, itwill be appreciated that the description of the operating subject ofeach step may be omitted.

First, learning data used to learn a driving pattern will be describedwith reference to FIG. 14 .

As shown in FIG. 14 , learning data 69 may include image data 63obtained by capturing surrounding environments of the mobile robot 30and a remote control value corresponding to the surrounding environmentsAccordingly, when a driving pattern (e.g., a degree of deceleration whenan object appears, a degree of deceleration on a curved road, etc.) of acorresponding user is learned through machine learning of the learningdata 69, a pattern control value of a corresponding user for a specificsurrounding environment may be predicted.

Also, the learning data 69 may further include information regarding arisk of accident 65, a level of complexity 67, and the like, which arecalculated based on the image data 63. The method of calculating therisk of accident 65 and the level of complexity 67, and thus redundantdescription thereof will be omitted.

Subsequently, a machine learning-based driving pattern learning methodwill be described with reference to FIG. 15 .

The driving pattern learning method may include an extraction process inwhich surrounding environment features 71 are extracted from the imagedata 63 included in the learning data 69 and a learning process in whichmachine learning is performed on pieces of learning data 61, 65, and 67other than the surrounding environment features 71.

The extraction process may be a process of extracting varioussurrounding environment features 71 through analysis of the image data63, and the surrounding environment features 71 may include variousfeatures such as the presence of lanes, an object recognition result,the presence of intersections, the presence of curved roads, thepresence of traffic lights, and the like. FIG. 15 shows an example inwhich various surrounding environment features 71 are extracted from theimage data 63 using a pre-learned convolutional neural network (CNN),but the technical scope of the present disclosure is not limitedthereto. For example, the surrounding environment features 71 may beextracted using a computer vision algorithm well-known in the art.

The learning process is a process of performing machine learning on thesurrounding environment features 71, the risk of collision 65, the levelof complexity 67, and the remote control value 61. In this case, thesurrounding environment features 71, the risk of collision 65, the levelof complexity 67, and the like may be used as input data of the machinelearning model, and the remote control value 61 may be used to updateweights of the machine learning model through comparison to a predictedcontrol value 73 of an output layer. FIG. 16 shows an example in whichmachine learning is performed using a deep learning model based on anartificial neural network (ANN), but the technical scope of the presentdisclosure is not limited thereto. Different types of machine learningalgorithms may be used to learn a driving pattern.

The learning of driving patterns of a plurality of users is merely thatthe learning data is expanded to learning data for the plurality ofusers, and thus redundant description thereof will be omitted. Dependingon the embodiment, it will be appreciated that a plurality of machinelearning model that learn driving patterns of individual users may bebuilt and also a single machine learning model that learns drivingpatterns of a plurality of users may be built. When the plurality ofmachine learning models are built, a pattern control value for aplurality of users may be determined based on an average, a weightedaverage, and the like of pattern control values of individual users. Inthis case, for example, accuracy, learning maturity, and the like ofeach machine learning model may be used as weights to be used for theweighted average.

The machine-learning-based driving pattern learning method according tosome embodiments of the present disclosure has been described withreference to FIGS. 14 and 15 .

The robot control system, the control method for the mobile robot, andthe control apparatus 100 for supporting the control method have beendescribed with reference to FIGS. 3 to 15 . A delivery system using amobile robot and an operating method for the system according to someembodiments of the present disclosure will be described below withreference to FIG. 16 and subsequent drawings. The aforementioned controlconcept for the mobile robot may be applied to a mobile robot which willbe described below.

FIG. 16 is a diagram illustrating a delivery system using a mobile robotaccording to some embodiments of the present disclosure.

As shown in FIG. 16 , the delivery system may include a service system200 and a mobile robot responsible for delivery (hereinafter referred toas a “delivery robot 300”). However, it should be appreciated that thisis merely an example embodiment for achieving the objects of the presentdisclosure and some elements are added to, or deleted from, theconfiguration as needed.

A brief overview of each element will be given. The service system 200is a computing system for providing a mobile robot-based deliveryservice to a user. The service system 200 may perform miscellaneousfunctions such as reception of a delivery request, allocation of adelivery mission according to a delivery request, and monitoring of adelivery status.

In some embodiments, as shown in FIG. 17 , the service system 200 mayinclude a service server 210, a control system 230, and a remote controlsystem 250.

The service server 210 is a computing apparatus that provides a webpagefor a delivery request or that performs all functions associated withthe delivery service, for example, management of a delivery history. Theservice server 210 may receive a delivery request from a user terminal400 and may request the control system 230 to generate a missioncorresponding to the delivery request.

Subsequently, the control system 230 is a system that performs allcontrol functions for the delivery robot 300, such as generation of adelivery mission, allocation of a delivery mission, and monitoring adelivery robot. The control system 230 may perform real-timecommunication with the delivery robot 300 and performing various kindsof monitoring and control functions. As shown in FIG. 17 , the controlsystem 230 may be implemented to include a plurality of displays tomonitor a plurality of delivery robots 300.

Subsequently, the remote control system 250 is a system having a remotecontrol function for the delivery robot 300. In some embodiments,through the remote control system 250, an operator may directly controlthe delivery robot 300 and provide the delivery service. The remotecontrol system 250 has been described with reference to FIG. 3 or thelike.

Referring to FIG. 16 again, the delivery robot 300 is a robot thatdelivers an item to be delivered (hereinafter referred to as a deliveryitem). The delivery robot 300 may autonomously perform delivery based onmission information and may also perform delivery under the control ofthe remote control system 250. As described above, it will beappreciated that the delivery robot 300 may be controlled by acombination of an autonomous control value and a remote control value ofthe remote control system 250.

The mission information includes all information needed by the deliveryrobot 300 to perform delivery. For example, the mission information mayinclude delivery origin information, delivery start time information,delivery destination information, target arrival time information,collaboration information for performing collaboration between deliveryrobots, information about a route to a delivery destination, etc., butthe technical scope of the present disclosure is not limited thereto.

The collaboration information includes all information needed to performcollaboration. For example, the collaboration information may includeinformation regarding roles (e.g., delivery, peripheral monitoring,etc.) of the delivery robots, information regarding transfer places andtransfer methods for items, and the like. The examples of thecollaboration information will be additionally described below alongwith an example in which collaborative delivery is performed.

The delivery robots 300 that perform the collaboration may be configuredin various forms. For example, a ground robot and a drone maycollaborate to perform delivery, and a plurality of ground robots or aplurality of drones may collaborate to perform delivery.

Subsequently, the user terminal 400 is a terminal that is used by a userto make a delivery request. The user terminal 400 may be implementedwith any device.

The delivery system using the mobile robot has been simply describedwith reference to FIGS. 16 and 17 . A mobile robot-based deliveryservice providing method performed by the delivery system will bedescribed below with reference to FIG. 18 .

FIG. 18 is an example flowchart showing a mobile robot-based deliveryservice providing method performed by a delivery system according tosome embodiments of the present disclosure. However, it should beappreciated that this is merely an example embodiment for achieving theobjects of the present disclosure and some steps are added to, ordeleted from, the configuration as needed.

As shown in FIG. 18 , the delivery service providing method begins withstep S510 in which a delivery request is received. As described above,the service server 210 may receive the delivery request.

In step S520, the service system 200 generates mission information fordelivery in response to the delivery request. As described above, thecontrol system 230 may designate a robot to be responsible for adelivery mission from among a plurality of delivery robots 300 and maygenerate mission information for the designated robot.

In step S530, the service system 200 transmits the generated missioninformation to the delivery robot 300 and allocates the delivery missionto the delivery robot 300,

In step S540, the delivery robot 300 starts the delivery based on thereceived mission information. For example, based on a delivery starttime and a delivery origin included in the mission information, thedelivery robot 300 may move to the delivery origin at the delivery starttime and load a delivery item to start the delivery.

In step S550, the delivery robot 300 periodically or aperiodicallyreports its own location information. Thus, a manager may monitor thelocation of the mobile robot 300 through the control system 230 in realtime.

Also, the delivery robot 300 may report, to the service system 200,information regarding various kinds of events (e.g., item damage,delivery delay, etc.) and information regarding surrounding environments(e.g., images of surrounding environments, information regardingidentified obstacles, etc.) which are measured through sensors.

In step S560, the service system 200 provides delivery statusinformation, location information, and the like of the delivery robot300 to the user terminal 400 in real time. Thus, it is possible toprovide a more satisfactory delivery service.

In steps S570 and S580, the delivery robot 300 arrives at a deliverydestination according to the received mission information and providesan arrival notification to the user. The movement to the deliverydestination may be performed by route information included in themission information and may also be performed by an autonomous drivingfunction of the delivery robot 300.

The arrival notification may be performed in any manner, including avisual message, such as flashing light, an audible message such as soundand music, a text message, and a phone call.

In some embodiments, as shown in FIG. 19 , the delivery robot 301 mayperform a terminal location identification process (303) and a useridentification process (305) before performing an arrival notification.In detail, the delivery robot 301 may estimate an approximate locationof a user terminal 401 using the strength (e.g. Received Signal StrengthIndicator (RSSI)) of signals received from the user terminal 401. Also,in the vicinity of the estimated location, the delivery robot 301 mayanalyze a captured image and recognize a user 403 by face. Then, thedelivery robot 301 may move to the vicinity of the recognized user 403and provide a delivery notification. Here, an image of the face of theuser 403 to be used for face recognition, identification information foridentifying the user terminal 401, a scheme for the deliverynotification, and the like may be pre-defined in the missioninformation.

The mobile robot-based delivery service providing method has beendescribed with reference to FIGS. 18 and 19 . According to theaforementioned method, by providing a quick and accurate deliveryservice using a mobile robot, it is possible to improve customersatisfaction and to increase delivery efficiency through reduced laborcosts.

According to some embodiments of the present disclosure, the deliveryrobot 300 includes a ground robot and a drone, and delivery may beperformed through collaboration between the ground robot and the drone.Thus, it is possible to provide a door-to-door delivery service or toexpand a delivery range. Various embodiments associated withcollaborative delivery will be described below with reference to FIGS.20 to 27 .

FIG. 20 is an example diagram illustrating a collaborative deliverymethod according to a first embodiment of the present disclosure.

As shown in FIG. 20 , according to the first embodiment, a drone 311 isresponsible for monitoring surrounding environments of a ground robot313 in the air, and the ground robot 313 is responsible for transportinga delivery item 315. Such role assignment may be defined incollaboration information for the drone 311 and the ground robot 313.

In more detail, the ground robot 313 transports the delivery item 315 toa deliver destination 317, and the drone 311 captures the surroundingenvironments of the ground robot 313 and provides monitoring informationsuch as obstacle information, topographic information, and trafficinformation to the ground robot 313. Then, the ground robot 313 maycorrect a route based on the provided monitoring information. Forexample, the ground robot 313 may reset the route to be a route foravoiding an obstacle or for maintaining a flat terrain. Thus, it ispossible to provide a safer delivery service. As another example, theground robot 313 may reset the route to be a low traffic route. Thus, itis possible to provide a quick delivery service.

FIG. 21 is an example diagram illustrating a collaborative deliverymethod according to a second embodiment of the present disclosure.

As shown in FIG. 21 , according to the second embodiment, both of aground robot 324 and a drone 326 are responsible for transport. Forexample, the ground robot 324 may be responsible for transporting adelivery item 325 from an origin 321 to an intermediate destination 322,and the drone 326 may be responsible for transporting the delivery item325 from the intermediate destination 322 to a delivery destination 323.In this case, location information (e.g., a transfer place) of theintermediate destination 322 and information regarding transfer timesand the like may be pre-defined in the collaboration information and maybe dynamically determined depending on the situation. In order todynamically perform collaboration, event information (hereinafterreferred to as a “collaboration event”) causing the collaboration may bedefined in the collaboration information.

In some embodiments, the collaboration event may occur when the groundrobot 324 or the drone 326 can no longer proceed with the transport. Forexample, the collaboration event may occur when an abnormality occurs inthe robot or when movement to a destination is no longer possible due toan obstacle or the like. In this case, a place where the collaborationevent occurs may be regarded as a transfer place (i.e., the intermediatedestination 322). An example in which the collaboration event occurs dueto an obstacle will be described with reference to FIG. 22 .

In some embodiments, the collaboration event may occur based on a riskof accident. The risk of accident will be described below with referenceto FIG. 31 .

FIGS. 22 and 23 are example diagrams illustrating a collaborativedelivery method according to a third embodiment of the presentdisclosure.

As shown in FIGS. 22 and 23 , according to the third embodiment, aground robot 331 collaborates with a drone 333 in response torecognition of an obstacle 337 present on a route. Like theaforementioned first embodiment, it will be appreciated that the dronemay detect or identify the obstacle 337 and provide informationregarding the obstacle 337 to the ground robot 331.

In this embodiment, the ground robot 331 may determine whether to bypassthe obstacle 337 based on the information regarding the obstacle 337.Here, the obstacle bypassing may include passing over the obstacle,avoiding the obstacle, and the like. Also, as shown in FIG. 23 , theground robot 331 may transfer a delivery item 335 to the drone 333 inresponse to determination that the obstacle 337 cannot be bypassed.

Here, the delivery item 335 may be transferred in any manner. Forexample, the ground robot 331 may transport the delivery item 335 whilemounting the drone 333 and may activate the drone 333 when the obstacle337 appears. It will be appreciated that the ground robot 331 mayreport, to the control system 230, that the obstacle 337 cannot bebypassed, and the drone 333 may be activated under the control of thecontrol system 230. As another example, the ground robot 331 may call adrone located in the vicinity and may transfer the delivery item 335 tothe drone.

In some embodiments, the ground robot 331 may determine whether tobypass a recognized obstacle 337 based on the size of the obstacle 337and the fragility of the delivery item 335. In this case, the fragilityrefers to a degree to which the item is devalued due to damage, and maybe determined to be a higher value as the item is of more substantialvalue (e.g., an expensive item), is fragile by its characteristics(e.g., an item made of a fragile material), and/or is greatlydepreciated due to damage. In detail, for example, when a fragiledelivery item is being transported, the ground robot 331 may determinethat a recognized obstacle cannot be bypassed even though the obstacleis small (i.e., even though the ground robot 331 can sufficiently bypassthe obstacle according to the performance of the robot). Thus, it ispossible to provide a safer delivery service for a fragile item.

FIGS. 24 and 25 are example diagrams illustrating a collaborativedelivery method according to a fourth embodiment of the presentdisclosure.

As shown in FIGS. 24 and 25 , according to the fourth embodiment, adoor-to-door delivery service may be provided through a drone (e.g.,341) and a ground robot (e.g., 345 to 347). In detail, air transport orinter-roof transport may be performed through the drone 341, and indoordelivery to the user's door may be accomplished through a ground robot345 located on the rooftop of a delivery destination 348 (e.g., atransfer place).

FIG. 25 shows a process of transferring a delivery item 353 to atransfer place (e.g., the rooftop of a delivery destination building348).

As shown in FIG. 25 , a marker 355 for identifying a drone 351 may bepre-installed at a transfer place (e.g., a rooftop), and the location,shape, and the like of the marker 355 may be pre-defined according tomission information.

The drone 351 may identify the installed marker 355 through imagecapturing and drop the delivery item 353 in an area where the marker 355is installed. A safety device may be pre-installed in the marker area toprevent the delivery item from being damaged. Subsequently, the drone351 transmits drop location information (i.e., location information ofthe marker 355 area) to the ground robot 357. Then, the ground robot 357may move to the location and receive the delivery item 353.

As reference, when a collaborative partner is not the ground robot 357but a first drone, the drone 351 may transmit location information ofthe marker area to the first drone. Then, the first drone may receivethe delivery item 353 and transport the delivery item 353 to thedelivery destination. It will be appreciated that the first drone mayalso transfer the delivery item 353 to another ground robot.

Also, the aforementioned embodiment may be applied even to a case inwhich collaboration is performed between ground robots withoutsubstantially changing the technical spirit. For example, when a firstground robot places a delivery item in a marker area and then transmitslocation information of the marker area to a second ground robot, thesecond ground robot may receive the delivery item using the locationinformation of the marker area.

The drop location information may be GPS information of the marker 355area, image information obtained by capturing the marker 355 area, orthe like, but the technical scope of the present disclosure is notlimited thereto.

FIG. 26 is an example diagram illustrating a collaborative deliverymethod according to a fifth embodiment of the present disclosure.

As shown in FIG. 26 , the fifth embodiment relates to a collaborativedelivery method for providing a door-to-door delivery service. In thisembodiment, a ground robot 361 transfers a delivery item 363 to a drone367 in a transfer place 365, and the drone 367 delivers the deliveryitem 363 to a user 369.

As shown in FIG. 26 , the drone 367 may approach a place of residence ofthe user 369 in a delivery destination building 366 to deliver thedelivery item 363. Height information (e.g., number of floors) of theresidence place may be defined in mission information.

In some embodiments, the drone 367 may provide a delivery notificationto the user 369 by continuously generating a visual or audio message inthe vicinity of the delivery destination building 366. Alternatively,the drone 367 and/or the service system 200 may provide a deliverynotification by transmitting a message to a terminal of the user 369.

In some embodiments, the drone 367 may identify the user 369 in a wayshown in FIG. 19 and may provide an arrival notification only when theuser 369 is identified.

FIG. 27 is an example diagram illustrating a collaborative deliverymethod according to a sixth embodiment of the present disclosure.

As shown in FIG. 27 , according to the sixth embodiment, a drone 375provides a delivery service in collaboration with a transport vehicle371 (e.g., a parcel service vehicle). FIG. 27 shows an example in whichthe transport vehicle 371 provides a delivery service in collaborationwith the drone 375, but the transport vehicle 371 may collaborate with adelivery robot (e.g., a ground robot) other than the drone 375.

As shown in FIG. 27 , when the transport vehicle 371 transports adelivery item to a designated transfer place 373, the drone 375 maytransport the delivery item to a delivery destination 377. In someembodiments, the drone 375 may provide a door-to-door delivery service,as shown in FIG. 26 .

In some embodiments, the delivery destination 377 may refer to a placethat the transport vehicle 371 cannot enter for various reasons. Forexample, the delivery destination 377 may refer to a place that thetransport vehicle 371 cannot enter for geographical reasons (e.g.,mountainous areas, island areas, and the like when an access road isnarrow or absent) or legal reasons (e.g., areas that are legallyrestricted by vehicular access). Even in this case, according to thisembodiment, the drone 375 may be utilized to accomplish delivery to thedelivery destination 377. Accordingly, it is possible to greatly expandthe delivery range.

The collaborative delivery methods according to various embodiments ofthe present disclosure have been described with reference to FIGS. 20 to27 . According to the aforementioned method, it is possible to provide aquick and safe delivery service through collaboration between a groundrobot and a drone. In particular, the delivery range may be greatlyexpanded by performing delivery to a destination through a drone eventhough a ground robot cannot reach the destination. Furthermore, it ispossible to significantly improve a customer's satisfaction by providinga door-to-door delivery service using a ground robot and a drone.

A method of safely delivering a large item in collaboration with aplurality of drones will be described below with reference to FIGS. 28and 29 .

FIG. 28 is an example diagram illustrating a collaborative deliverymethod based on multiple drones according to a first embodiment of thepresent disclosure.

As shown in FIG. 28 , the first embodiment relates to a method of safelydelivering a large delivery item 387 using transmission ropes of theplurality of drones 381, 373, and 385. The large delivery item 387 mayrefer to an item with a weight greater than or equal to a thresholdvalue, but the technical scope of the present disclosure is not limitedthereto.

In order to safely deliver the large delivery item 387, it is importantthat the center of gravity of the large delivery item 387 is kept lowwhile the drones 381, 383, and 385 are moving (i.e., it is importantthat the orientation of the large delivery item is kept balanced). Tothis end, each of the drones 381, 383, and 385 estimates the orientationof the large delivery item 387 based on various factors and determineswhether the estimated orientation satisfies a predetermined balancecondition. Also, each of the drones 381, 383, and 385 may adjust thelength of the transmission rope or its relative location in response todetermination that the balance condition is not satisfied.

In some embodiments, the factors may include the length of thetransmission rope, the relative location of the drone, the direction ofthe transmission rope, and the like.

In some embodiments, the balance condition may be, for example, acondition based on a degree to which the large delivery item 387 istilted (e.g., the large delivery item 387 is parallel to a level surfacewhen the degree is 0), but the technical scope of the present disclosureis not limited thereto. Also, the balance condition may be more strictlyset as the fragility of the delivery item 387 increases, but thetechnical scope of the present disclosure is not limited thereto.

FIG. 29 is an example diagram illustrating a collaborative deliverymethod based on multiple drones according to a second embodiment of thepresent disclosure.

As shown in FIG. 29 , according to the second embodiment, a specificdrone 391 that does not participate in transport operates as acoordinator. In more detail, the coordinator drone 391 captures imagesof a large delivery item 399 and drones 393, 395, and 397 fortransporting the large delivery item 399, analyzes the captured images,and measures the orientation of the large delivery item 399 and relativelocations of the transport drones 393, 395, and 397.

Also, the coordinator drone 391 transmits an adjustment command to thespecific drone (e.g., 395) in order to correct the orientation of thelarge delivery item 399 in response to determination that theorientation of the large delivery item 399 does not satisfy thepredetermined balance condition. In this case, the adjustment commandmay include commands regarding adjustment of the location of the drone,adjustment of the length of the rope, and adjustment of the direction ofthe rope.

In this embodiment, the coordinator drone 391 may continuously monitorthe transport drones 393, 395, and 397 and the large delivery item 399and may perform continuous adjustment so that the orientation of thelarge delivery item 399 is kept balanced. Thus, it is possible toprovide a more safe delivery service.

The method of safely delivering a large item in collaboration with aplurality of drones has been described with reference to FIGS. 28 and 29. According to the aforementioned method, it is possible to deliver alarge item utilizing a plurality of drones, and it is also possible toprovide a safe delivery service through orientation correction.

Various embodiments of the present disclosure for improving a customer'ssatisfaction while a food delivery service is provided will be describedbelow with reference to FIGS. 30 to 32 .

The following embodiments relate to a case in which food is delivered.In such an embodiment, a service system 200 may receive a deliveryrequest (e.g., food order) from a user terminal 400 and may generatemission information for delivering food to a delivery destination inresponse to the request. Also, a mobile robot for delivering food(hereinafter referred to as a “delivery robot 405”) may deliver the foodbased on the mission information. Hereinafter, various embodiments ofthe present disclosure will be described in detail with reference toFIGS. 30 to 32 .

FIG. 30 is an example diagram illustrating a food delivery method usinga mobile robot according to a first embodiment of the presentdisclosure.

As shown in FIG. 30 , according to the first embodiment, the deliveryrobot 405 delivers food with a cooking appliance containing the food.The delivery robot 405 performs food delivery based on missioninformation generated by the service system 200. Here, the missioninformation includes a cooking time for the food.

In this embodiment, while moving from an origin 401 to a destination403, the delivery robot 405 calculates an expected time taken to reachthe destination 403. The expected time may be calculated in any way.Also, the delivery robot 405 compares the cooking time for the foodincluded in the mission information to the expected time and operatesthe cooking appliance 407 based on a result of the comparison. Forexample, when the expected time is the same as the cooking time, thedelivery robot operates the cooking appliance 407 so that the food maybe appropriately cooked when the destination 403 is reached. Thus, it ispossible to provide an optimal food delivery service to a customer.

When a separate cooking time is not designated, the delivery robot 405may deliver the food while activating the cooking appliance 407 in awarming or cooling state so that the taste or freshness of the food canbe maintained.

The cooking appliance 407 refers to a tool having various cookingfunctions. For example, the cooking appliance 407 may be a tool havingvarious cooking functions such as heating, cooling, frying, warming,cooking, and baking. Accordingly, the delivery robot 405 may cook foodusing the cooking appliance 407 (e.g., warming pizza, frying fritters,boiling stew, steaming rice, or baking pizza in an oven) whiledelivering the food, and the customer may receive the just-cooked food.

As surrounding situations (e.g., a traffic volume) change, an actualtime taken to reach the destination 403 may differ from the expectedtime. For example, the expected time may be longer or shorter than theactual time. In this case, the delivery robot 405 may adjust a cookingcompletion time of the food to an arrival time by adjusting cookingintensity (e.g., heating intensity) of the cooking appliance 407.

In more detail, the delivery robot 405 may periodically update theexpected time taken to reach the delivery destination 403 and may adjustthe cooking intensity of the cooking appliance 407 according to theupdated expected time. For example, when the updated expected time islonger than before, the delivery robot 405 may adjust the cookingintensity of the cooking appliance 407 to be low. Otherwise, thedelivery robot 405 may adjust the cooking intensity of the cookingappliance 407 to be high.

In some embodiments, the delivery robot 405 may monitor a cooking statusof the contained food by using a sensor included in the cookingappliance 407. For example, an image sensor, a temperature sensor, andthe like may be used to monitor the cooking status of the food. Also,the delivery robot 405 may adjust the cooking intensity of the cookingappliance 407 based on the monitored cooking status. For example, thecooking intensity may be adjusted to be high or low depending on thecooking status.

Also, in some embodiments, the delivery robot 405 may sense surroundingenvironments and calculate a risk of accident for a current route basedon the sensing data. Also, the delivery robot 405 may adjust the routeto the delivery destination to another route in response todetermination that the calculated risk of accident is greater than orequal to a threshold value. This is because food may be more perishablethan other delivery items. Thus, it is possible to provide a safedelivery service to the destination 403.

The threshold value may be a predetermined constant or a variablevarying depending on situations. For example, the threshold value may bea variable determined based on the perishability of the contained food.Here, the perishability refers to a degree to which the food is devalueddue to damage, and may be determined to be a higher value as the food isof more substantial value (e.g., expensive food), is more perishable byits characteristics (e.g., perishable food such as pizza toppings),and/or is greatly depreciated due to damage. In more detail, forexample, when perishable food is being transported, the perishabilitymay be determined to be a value lower than the threshold value so thatthe delivery is performed through a safer route.

The specific method of calculating the risk of accident may varydepending on the embodiment.

In some embodiments, the risk of accident may be calculated based on atleast one of the curvature and slope of a road ahead. For example, ahigher value may be calculated as the risk of accident as the curvatureor slope of the road increases.

In some embodiments, the risk of accident may be calculated based on aresult of recognizing an object located within a certain distance. Theobject recognition result may be acquired through image analysis. Here,the object recognition result may include the number of moving objectsand a degree of irregularity of movement of an object. For example, asstill another example, as the number of mobile objects showing irregularmovement patterns increases, the level of complexity may be determinedto be a higher value.

In some embodiments, the risk of accident may be calculated in a mannersimilar to those of the risk of collision and the level of complexity ofsurrounding environments. The foregoing should be referred to by themethod of calculating the risk of collision and the method ofcalculating the level of complexity of surrounding environments.

In some embodiments, the risk of accident may be calculated based on acombination of the aforementioned embodiments.

Also, according to some embodiments of the present disclosure, theground robot 405 may transfer the cooking appliance 407 to a drone inresponse to determination that the risk of accident is greater than orequal to a threshold value. Thus, through the drone, it is possible toaccomplish a safe delivery, and it is also possible to provide adoor-to-door delivery service. Depending on the embodiment, it will beappreciated that a drone may transfer a cooking appliance to a groundrobot. For example, when surrounding environments are complex so thatflight is difficult (i.e., when the risk of accident is high), a dronemay transfer a cooking appliance to a ground robot.

FIGS. 31 and 32 are example diagrams illustrating a food delivery methodusing a mobile robot according to a second embodiment of the presentdisclosure.

As shown in FIGS. 31 and 32 , the second embodiment relates to a methodof providing a safe delivery service by adjusting the orientation of adeliver container (e.g., a cooking appliance) with food.

In the second embodiment, delivery robots 411 and 421 may adjust theorientation of the delivery container while moving according to a firstadjustment scheme when the food is of a first type and may adjust theorientation of the delivery container while moving according to a secondadjustment scheme when the food is of a second type. That is, dependingon the type of food, it is possible to accomplish the safe delivery withoptimal orientation.

Here, the type of the food may be classified into liquid food such assoup and stew and non-liquid food such as pizza and steamed rice, butthe technical scope of the present disclosure is not limited thereto.

FIG. 31 illustrates a method of adjusting the orientation of a deliverycontainer with respect to liquid food.

As shown in FIG. 31 , when a delivery robot 411 is accelerated, thesurface of the food may be sloped due to reaction acceleration, and thusthe liquid food may spill over from the delivery container 413. In orderto prevent such a problem, the delivery robot 411 may adjust theorientation of the delivery container 413 to cancel the reactionacceleration.

In more detail, the delivery robot 411 may measure the surface slope ofthe food in the delivery container 413 and may adjust the orientation ofthe delivery container 413 to lower the surface slope. The measurementof the surface slope may be performed through image analysis, but thetechnical scope of the present disclosure is not limited thereto. Forexample, when the surface slope occurs due to the reaction acceleration,the delivery robot 411 may tilt the delivery container 413 so that thesurface of food is parallel to a level surface, as shown in FIG. 31 .Thus, it is possible to prevent the liquid food from spilling over fromthe delivery container 413.

In some embodiments, the delivery robot 411 may measure drivingacceleration and may adjust the orientation of the delivery container413 based on the magnitude of the driving acceleration. For example, thereaction acceleration may increase as the driving accelerationincreases. Accordingly, the delivery robot 411 may adjust theorientation of the delivery container 413 to further tilt the deliverycontainer 413 (i.e., so that the surface of the food is parallel to ahorizontal plane).

As reference, FIG. 31 shows an example in which the delivery robot 411is accelerated on a flat road. However, it should be noted that theaforementioned technical spirit may be applied to a case in which a roadis tilted or curved or a case in which a drone is accelerated in theair. In any case, the delivery robot 411 may adjust the orientation ofthe delivery container 413 to lower the surface slope of the foodcontained in the delivery container 413.

FIG. 32 illustrates a method of adjusting the orientation of a deliverycontainer with respect to non-liquid food.

As shown in FIG. 32 , a delivery robot 421 may measure a ground slopeand may adjust the orientation of the delivery container 423 based onthe ground slope. For example, when the delivery robot 421 is moving ina sloped terrain, the delivery robot 421 may adjust the orientation ofthe delivery container 423 so that the delivery container 423 or thefood contained in the delivery container 423 is parallel to a horizontalplane. Thus, it is possible to accomplish safe delivery whilemaintaining the appearance of food (e.g., while preventing pizzatoppings from shifting from one side to another.

As reference, FIG. 32 shows an example in which the delivery robot 421is traveling in a vertically sloped area. However, it should be notedthat the aforementioned technical spirit may be applied to a case inwhich a road is horizontally tilted or curved or a case in which a droneis accelerated in mid-air. In any case, the delivery robot 421 mayadjust the orientation of the delivery container 413 to keep theorientation of the delivery container 413 parallel to a horizontalsurface.

Various embodiments of the present disclosure associated with a fooddelivery service have been described with reference to FIGS. 30 to 32 .As described above, just-cooked food may be delivered to a customer bycooking while moving. Accordingly, it is possible to improve acustomer's satisfaction. Furthermore, it is possible to provide a safedelivery service by adjusting the orientation of the delivery containeraccording to the type of food. Accordingly, it is possible to furtherenhance a customer's satisfaction.

An example computing apparatus 500 capable of implementing apparatuses(e.g., 100, 210, and 300) and/or systems (e.g., 230 and 250) accordingto various embodiments of the present disclosure will be described belowwith reference to FIG. 33 .

FIG. 33 is an example hardware configuration diagram showing the examplecomputing apparatus 500.

As shown in FIG. 33 , the computing apparatus 500 may include at leastone processor 510, a bus 550, a communication interface 570, a memoryconfigured to load a computer program 591 performed by the processor510, and a storage 590 configured to store the computer program 591.However, only elements related to the embodiment of the presentdisclosure are shown in FIG. 33 . Accordingly, it is to be understood bythose skilled in the art that general-purpose elements other than theelements shown in FIG. 33 may be further included.

The processor 510 controls overall operation of each element of thecomputing apparatus 500. The processor 510 may be configured to includea central processing unit (CPU), a micro processor unit (MPU), a microcontroller unit (MCU), a graphic processing unit (GPU), or any type ofprocessor that is well known in the art. Also, the processor may performcomputation on at least one application or program for executing themethods according to the embodiments of the present disclosure. Thecomputing apparatus 500 may have one or more processors.

The memory 530 stores various kinds of data, commands, and/orinformation. The memory 530 may load the computer program 591 from thestorage 590 to perform the methods/operations according to variousembodiments of the present disclosure. The memory 530 may be implementedwith a volatile memory such as a random access memory (RAM), but thetechnical scope of the present disclosure is not limited thereto.

The bus 550 provides a communication function between the elements ofthe computing apparatus 500. The bus 550 may be implemented with varioustypes of buses such as an address bus, a data bus, and a control bus.

The communication interface 570 supports wired or wireless Internetcommunication of the computing apparatus 500. Also, the communicationinterface 570 may support various communication schemes other thanInternet communication. To this end, the communication interface 570 maybe configured to include a communication module that is well known inthe art.

The storage 590 may non-temporarily store one or more programs 591. Thestorage 590 may be configured to include a non-volatile memory such as aread only memory (ROM), an erasable programmable ROM (EPROM), anelectrically erasable programmable ROM (EEPROM), and a flash memory, ahard disk, a removable disk, or any computer-readable recording mediumthat is well know in the art.

The computer program 591 may include one or more instructions that allowthe processor 510 to perform the operations/methods according to variousembodiments of the present disclosures when the computer program 591 isloaded in the memory 530. The processor 510 may perform theoperations/methods according to various embodiments of the presentdisclosure by executing the one or more instructions.

For example, the computer program 591 may include instructions forperforming an operation of acquiring a remote control value for themobile robot 30, which is input through the remote control system 20, anoperation of acquiring an autonomous control value for the mobile robot30, which is generated by an autonomous driving module, an operation ofdetermining a weight for each control value based on a delay between themobile robot 30 and the remote control system 20, and an operation ofgenerating a target control value of the mobile robot 30 based on thedetermined weight, a first control value, and a second control value. Inthis case, the control apparatus 100 may be implemented through thecomputing apparatus 500.

The example computing apparatus 500 capable of implementing theapparatuses and/or systems according to various embodiments of thepresent disclosure has been described with reference to FIG. 33 .

So far, some embodiments of the present disclosure and the effectaccording to embodiments have been aforementioned with reference toFIGS. 1 to 33 . It should be noted that the effects of the presentdisclosure are not limited to the above-described effects, and othereffects will be apparent to those skilled in the art from the followingdescriptions.

The concepts of the disclosure described above with reference to FIGS. 1to 33 can be embodied as computer-readable code on a computer-readablemedium. The computer-readable medium may be, for example, a removablerecording medium (a CD, a DVD, a Blu-ray disc, a USB storage apparatus,or a removable hard disc) or a fixed recording medium (a ROM, a RAM, ora computer-embedded hard disc). The computer program recorded on thecomputer-readable recording medium may be transmitted to anothercomputing apparatus via a network such as the Internet and installed inthe computing apparatus. Hence, the computer program can be used in thecomputing apparatus.

Although operations are shown in a specific order in the drawings, itshould not be understood that desired results can be obtained when theoperations must be performed in the specific order or sequential orderor when all of the operations must be performed. In certain situations,multitasking and parallel processing may be advantageous. According tothe above-described embodiments, it should not be understood that theseparation of various configurations is necessarily required, and itshould be understood that the described program components and systemsmay generally be integrated together into a single software product orbe packaged into multiple software products.

While the present disclosure has been particularly illustrated anddescribed with reference to exemplary embodiments thereof, it will beunderstood by those of ordinary skill in the art that various changes inform and detail may be made therein without departing from the spiritand scope of the present disclosure as defined by the following claims.The exemplary embodiments should be considered in a descriptive senseonly and not for purposes of limitation.

What is claimed is:
 1. A collaborative delivery system using a mobilerobot, the collaborative delivery system comprising: a service systemconfigured to generate mission information for delivering an item to adelivery destination; a first mobile robot configured to transport theitem based on the mission information; and a second mobile robotconfigured to transport the item in collaboration with the first mobilerobot, wherein the mission information includes collaborationinformation necessary for the first mobile robot and the second mobilerobot to collaborate with each other, wherein the second mobile robotcontinuously monitors the first mobile robot and the item, and whereinthe second mobile robot transmits an adjustment command to the firstmobile robot in order to correct an orientation of the item in responseto determination that the orientation of the item does not satisfy apredetermined balance condition.
 2. The collaborative delivery system ofclaim 1, wherein the service system receives a delivery request for theitem from a user terminal, monitors a location of a mobile robot whichtransports the item, and provides the monitored location to the userterminal.
 3. The collaborative delivery system of claim 1, wherein thecollaboration information includes a transfer place for the item,wherein the first mobile robot is a drone, and the second mobile robotis a ground robot, wherein the first mobile robot recognizes a markerlocated in the vicinity of the transfer place and places the item in anarea of the recognized marker, and wherein the second mobile robotreceives the item placed in the marker area and transports the receiveditem.
 4. The collaborative delivery system of claim 3, wherein the firstmobile robot transmits image information obtained by capturing themarker area to the second mobile robot, and wherein the second mobilerobot receives the item placed in the marker area based on the imageinformation.
 5. The collaborative delivery system of claim 1, whereinthe first mobile robot is a ground robot, and the second mobile robot isa drone, and wherein the first mobile robot recognizes an obstaclepresent on a route to the destination and transfers the item to thesecond mobile robot in response to determination that the recognizedobstacle cannot be bypassed.
 6. The collaborative delivery system ofclaim 5, wherein the first mobile robot determines whether to bypass therecognized obstacle based on a size of the recognized obstacle and arisk of damage to the item.
 7. The collaborative delivery system ofclaim 1, wherein the first mobile robot and the second mobile robot aredrones, wherein the first mobile robot and the second mobile robottransport the item using ropes connected to the item, and wherein thefirst mobile robot estimates an orientation of the item based on alocation relative to the second mobile robot and a current length of therope and adjusts the length of the rope or the current location inresponse to determination that the estimated orientation does notsatisfy a predetermined balance condition.
 8. The collaborative deliverysystem of claim 1, wherein each of the first mobile robot and the secondmobile robot is a ground robot or a drone.
 9. The collaborative deliverysystem of claim 1, wherein the mission information includesidentification information of a user ordering the item, wherein thefirst mobile robot or the second mobile robot identifies the user byusing the identification information, and wherein the first mobile robotor the second mobile robot provides a message to the user in response todetermination that the user is identified.